The present invention relates to techniques for digitally capturing and processing images of a scene using an image detector, and more particularly, to techniques for correcting blurring introduced into such images due to motion of the image detector or due to movement of objects.
Motion blur due to camera shake is a common problem in photography, especially in conditions involving zoom and low light. Merely pressing a shutter release button on the camera can in and of itself cause the camera to shake, and unfortunately cause blurred images. This problem is especially prevalent in digital photography, where lightweight cameras with automated exposure times that are not easily stabilized are common, and where automated exposure times often necessitate relatively long stabilization periods to ensure a non blurred image. The compact form and small lenses of many of these cameras only serves to increase this problem.
The sensor of a digital camera creates an image by integrating energy over a period of time. If during this time—the exposure time—the image moves, either due to camera or object motion, the resulting image will exhibit motion blur. The problem of motion blur is increased when a long focal length (zoom) is employed, since a small angular change of the camera creates a large displacement of the image, and in situations when long exposure is needed, either due to lighting conditions, or due to the use of small aperture.
There are putative solutions to this problem. One group of solutions focuses on the reduction or elimination of relative movement between the camera and the scene during the integration or exposure time. Such solutions include the use of a tripod, flash photography, the use of increased sensor sensitivity, the use of an increased aperture, and dynamic image stabilization.
A stable tripod that can resist wind, and a shutter release cable that does not transmit hand vibration to a camera mounted on such a tripod, eliminates the problem of camera shake where both the mounted camera and scene are static. In practice only professionals are likely to use a tripod and shutter release cable on a regular basis. Moreover, the use of a tripod does not solve the problem of shooting from a moving platform, such as car, train, helicopter or balloon.
A photographic flash produces a strong light flux that sustained for a fraction of a section (less than 1/1000). The exposure time is set to bracket the flash time (usually 1/60 sec), and the aperture of the camera is set to match the flash flux. Therefore, blur caused by motion during the bracket time has very low intensity. In essence, the flash “freezes” motion of both camera and moving objects. However, objects in bright daylight may still have motion blur and, of course, flash photography is useful only if the distance between the flash and the object is small.
Increasing the sensor sensitivity, and therefore reducing the exposure time, can decrease the problem of motion blur. However, it cannot eliminate blur completely. However, sensitive sensors (whether film or CCD) produce noisy and grainy images.
Increasing the aperture size greatly decreases the required exposure time, and thus reduces motion blur. Unfortunately, the cost and weight of a camera also increase significantly with an increased lens aperture, and further a tripod may be required to comfortably handle such increased weight. Also, the use of a larger aperture lens is applicable only for more expensive cameras where it is possible to replace the lens.
In addition, the use of dynamic image stabilization involves the incorporation of inertial sensors, such as gyroscopes, to sense and compensate for camera shake in real time by moving an optical element. While this technology can be used in stabilized zoom lens for Single Lens Reflex (“SLR”) cameras, it is costly, and its effectiveness is limited to approximately 1/60 of a second for typical 400 mm zoom lens. The sensitivity of such system to very slow motion may also be limited, and may suffer from drift. In addition, such system cannot compensate for constant speed motion, such as occurs when taking images from a moving train.
Accordingly, while addressing the problem of motion of the camera itself is useful in certain applications, it does not lead to an adequate solution to the problem of motion blur as such systems are either limited, very costly, or both. An alternative approach is to correct blur after the image has been taken by using a de-blurring algorithm.
Co-invented and co-assigned International patent application WO 2004001667 describes hybrid systems and methods for deblurring a motion blurred image of a scene. The motion blurring may, for example, be due to camera or detector motion. The described hybrid systems and methods involve deploying a secondary image detector in addition to the primary detector that senses the motion blurred primary image of a scene. The secondary detector is deployed to sense additional or secondary images of the scene. An image processor estimates a point spread function based on the secondary images to characterize the motion seen in the primary image. This point spread function may be applied globally to the motion blurred primary image to generate a de-blurred image.
Images may also be blurred because the scene includes moving objects. A scene may include a stationary background through which objects move. The movement of an object in a scene may blur portions of the recorded images corresponding to the moving object in addition to any image blurring due to camera motion. A global transformation of an image cannot remove the relative motion of the object and the background in the scene. Thus, point spread functions that are applied globally to a motion blurred primary image to correct for camera motion are ineffective in deblurring motion blur caused by moving objects internal to the scene.
Further consideration is now being given to systems and methods for de blurring motion blurred primary images. In particular attention is directed to deblurring images of scenes that involve moving objects.